Dataflow methods in HPC, visualisation and analysis

The processing power available to scientists and engineers using supercomputers over the last few decades has grown exponentially, permitting significantly more sophisticated simulations, and as a consequence, generating proportionally larger output datasets. This change has taken place in tandem with a gradual shift in the design and implementation of simulation and post-processing software, with a shift from simulation as a first step and visualisation/analysis as a second, towards in-situ on the fly methods that provide immediate visual feedback, place less strain on file-systems and reduce overall data-movement and copying. Concurrently, processor speed increases have dramatically slowed and multi and many-core architectures have instead become the norm for virtually all High Performance computing (HPC) machines. This in turn has led to a shift away from the traditional distributed one rank per node model, to one rank per process, using multiple processes per multicore node, and then back towards one rank per node again, using distributed and multi-threaded frameworks combined. This thesis consists of a series of publications that demonstrate how software design for analysis and visualisation has tracked these architectural changes and pushed the boundaries of HPC visualisation using dataflow techniques in distributed environments. The first publication shows how support for the time dimension in parallel pipelines can be implemented, demonstrating how information flow within an application can be leveraged to optimise performance and add features such as analysis of time-dependent flows and comparison of datasets at different timesteps. A method of integrating dataflow pipelines with in-situ visualisation is subsequently presented, using asynchronous coupling of user driven GUI controls and a live simulation running on a supercomputer. The loose coupling of analysis and simulation allows for reduced IO, immediate feedback and the ability to change simulation parameters on the fly. A significant drawback of parallel pipelines is the inefficiency caused by improper load-balancing, particularly during interactive analysis where the user may select between different features of interest, this problem is addressed in the fourth publication by integrating a high performance partitioning library into the visualization pipeline and extending the information flow up and down the pipeline to support it. This extension is demonstrated in the third publication (published earlier) on massive meshes with extremely high complexity and shows that general purpose visualization tools such as ParaView can be made to compete with bespoke software written for a dedicated task. The future of software running on many-core architectures will involve task-based runtimes, with dynamic load-balancing, asynchronous execution based on dataflow graphs, work stealing and concurrent data sharing between simulation and analysis. The final paper of this thesis presents an optimisation for one such runtime, in support of these future HPC applications

Data Redistribution using One-sided Transfers to In-memory HDF5 Files

International audienceOutputs of simulation codes making use of the HDF5 file format are usually and mainly composed of several different attributes and datasets, storing either lightweight pieces of information or containing heavy parts of data. These objects, when written or read through the HDF5 layer, create metadata and data IO operations of different block sizes, which depend on the precision and dimension of the arrays that are being manipulated. By making use of simple block redistribution strategies, we present in this paper a case study showing HDF5 IO performance improvements for "in-memory" files stored in a distributed shared memory buffer using one-sided communications through the HDF5 API

Visualization and analysis of SPH data

Advances in graphics hardware in recent years have led not only to a huge growth in the speed at which 3D data can be rendered, but also to a marked change in the way in which different data types can be displayed. In particular, point based rendering techniques have benefited from the advent of vertex and fragment shaders on the GPU which allow simple point primitives to be displayed not just as dots, but rather as complex entities in their own right. We present a simple way of displaying arbitrary 2D slices through 3D SPH data by evaluating the SPH kernel on the GPU and accumulating the contributions from individual particles intersecting a slice plane into a texture. The resulting textured plane can then be displayed alongside the particle based data. Combining 2D slices and 3D views in an interactive way improves perception of the underlying physics and speeds up the development cycle of simulation code. In addition to rendering particles themselves, we can improve visualization by generating particle trails to show motion history, glyphs to show vector fields, transparency to enhance or diminish areas of high/low interest and multiple views of the same or different data for comparative visualization. We combine these techniques with interactive control or arbitrary scalar parameters and animation through time to produce a feature rich environment for exploration of SPH data

High performance computing 3D SPH model: Sphere impacting the free-surface of water

In this work, an analysis based on a three-dimensional parallelized SPH model developed by ECN and applied to free surface impact simulations is presented. The aim of this work is to show that SPH simulations can be performed on huge computer as EPFL IBM Blue Gene/L with 8'192 cores. This paper presents improvements concerning namely the memory consumption, which remains quite subtle because of the variable-H scheme constraints. These improvements have made possible the simulation of test cases involving tens of millions of particles computed by using more than thousand cores. Furthermore, pv-meshless developed by CSCS, is used to show the pressure field and the effect of impact

Advanced visualization of large datasets for Discrete Element Method simulations

State-of-the-art Discrete Element Method (DEM) simulations of granular flows produce large datasets that contain a wealth of information describing the time-dependent physical state of the particulate medium. To extract this information, both comprehensive and efficient post-processing methods are essential. Special attention must be paid to the interactive visualization of these large hybrid datasets containing both particle-based and surface-based data. In this paper, we report the use of the open-source visualization package ParaView, which we have customized specifically to perform advanced techniques for the post-treatment of large DEM datasets. Particular attention is given to the method used to render the individual particles, based either on triangulation of glyphs or using GPU-accelerated primitives. A demonstration of these techniques, and their relative merits when applied to the visualization of DEM datasets, is presented via their application to real industrial examples

Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver

Honourable Mention AwardInternational audienceWe present a framework for interfacing an arbitrary HPC simulation code with an interactive ParaView session using the HDF5 parallel IO library as the API. The implementation allows a flexible combination of parallel simulation, concurrent parallel analysis and GUI client, all of which may be on the same or separate machines. Data transfer between the simulation and the ParaView server takes place using a virtual file driver for HDF5 that bypasses the disk entirely and instead communicates directly between the coupled applications in parallel. The simulation and ParaView tasks run as separate MPI jobs and may therefore use different core counts and/or hardware configurations/platforms, making it possible to carefully tailor the amount of resources dedicated to each part of the workload. The coupled applications write and read datasets to the shared virtual HDF5 file layer, which allows the user to read data representing any aspect of the simulation and modify it using ParaView pipelines, then write it back, to be reread by the simulation (or vice versa). This allows not only simple parameter changes, but complete remeshing of grids, or operations involving regeneration of field values over the entire domain, to be carried out. To avoid the problem of manually customizing the GUI for each application that is to be steered, we make use of XML templates that describe outputs from the simulation, inputs back to it, and what user interactions are permitted on the controlled elements. This XML is used to generate GUI and 3D controls for manipulation of the simulation without requiring explicit knowledge of the underlying model

From Piz Daint to the Stars: Simulation of Stellar Mergers using High-Level Abstractions

We study the simulation of stellar mergers, which requires complex simulations with high computational demands. We have developed Octo-Tiger, a finite volume grid-based hydrodynamics simulation code with Adaptive Mesh Refinement which is unique in conserving both linear and angular momentum to machine precision. To face the challenge of increasingly complex, diverse, and heterogeneous HPC systems, Octo-Tiger relies on high-level programming abstractions. We use HPX with its futurization capabilities to ensure scalability both between nodes and within, and present first results replacing MPI with libfabric achieving up to a 2.8x speedup. We extend Octo-Tiger to heterogeneous GPU-accelerated supercomputers, demonstrating node-level performance and portability. We show scalability up to full system runs on Piz Daint. For the scenario's maximum resolution, the compute-critical parts (hydrodynamics and gravity) achieve 68.1% parallel efficiency at 2048 nodes.Comment: Accepted at SC1

Addressing the welfare needs of farmed lumpfish: knowledge gaps, challenges and solutions

Lumpfish (Cyclopterus lumpus L.) are increasingly being used as cleaner fish to control parasitic sea lice, one of the most important threats to salmon farming. However, lumpfish cannot survive feeding solely on sea lice, and their mortality in salmon net pens can be high, which has welfare, ethical and economic implications. The industry is under increasing pressure to improve the welfare of lumpfish, but little guidance exists on how this can be achieved. We undertook a knowledge gap and prioritisa tion exercise using a Delphi approach with participants from the fish farming sector, animal welfare, academia and regulators to assess consensus on the main challenges and potential solutions for improving lumpfish welfare. Consensus among participants on the utility of 5 behavioural and 12 physical welfare indicators was high (87–89%), reliable (Cronbach's alpha = 0.79, 95CI = 0.69–0.92) and independent of participant background. Participants highlighted fin erosion and body damage as the most use ful and practical operational welfare indicators, and blood parameters and behav ioural indicators as the least practical. Species profiling revealed profound differences between Atlantic salmon and lumpfish in relation to behaviour, habitat preferences, nutritional needs and response to stress, suggesting that applying a common set of welfare standards to both species cohabiting in salmon net-pens may not work well for lumpfish. Our study offers 16 practical solutions for improving the welfare of lumpfish and illustrates the merits of the Delphi approach for achieving consensus among stakeholders on welfare needs, targeting research where is most needed and generating workable solutions.info:eu-repo/semantics/publishedVersio

Determination of regional lung air volume distribution at mid-tidal breathing from computed tomography: A retrospective study of normal variability and reproducibility

© 2014 Fleming et al.; licensee BioMed Central Ltd. Background: Determination of regional lung air volume has several clinical applications. This study investigates the use of mid-tidal breathing CT scans to provide regional lung volume data.Methods: Low resolution CT scans of the thorax were obtained during tidal breathing in 11 healthy control male subjects, each on two separate occasions. A 3D map of air volume was derived, and total lung volume calculated. The regional distribution of air volume from centre to periphery of the lung was analysed using a radial transform and also using one dimensional profiles in three orthogonal directions.Results: The total air volumes for the right and left lungs were 1035 +/- 280 ml and 864 +/- 315 ml, respectively (mean and SD). The corresponding fractional air volume concentrations (FAVC) were 0.680 +/- 0.044 and 0.658 +/- 0.062. All differences between the right and left lung were highly significant (p < 0.0001). The coefficients of variation of repeated measurement of right and left lung air volumes and FAVC were 6.5% and 6.9% and 2.5% and 3.6%, respectively. FAVC correlated significantly with lung space volume (r = 0.78) (p < 0.005). FAVC increased from the centre towards the periphery of the lung. Central to peripheral ratios were significantly higher for the right (0.100 +/- 0.007 SD) than the left (0.089 +/- 0.013 SD) (p < 0.0001).Conclusion: A technique for measuring the distribution of air volume in the lung at mid-tidal breathing is described. Mean values and reproducibility are described for healthy male control subjects. Fractional air volume concentration is shown to increase with lung size.Air Liquid

Computational Steering and Parallel Online Monitoring Using RMA through the HDF5 DSM Virtual File Driver

International audienceAs systems provide more resources and host an indefinitely growing number of cores, the amount of data produced by simulation codes is steadily increasing, creating a bottleneck at the point where the data must be transferred to post-processing software. One solution is to avoid the use of a file system altogether and couple post processing software directly to the simulation using an interface common to both sides of the transfer. HDF5, the widely known IO library, offers a modular mapping of file contents to storage, allowing the user to use different methods (drivers) for reading and writing data. These drivers are organized in a Virtual File Layer (VFL) so that the user can easily switch between - and if necessary - extend them. In order to be able to visualize and analyze data in-situ, we developed a parallel virtual file driver called the DSM driver, which allows the transfer of data in parallel between two different codes using only the HDF5 API; this driver has now been extended to support remote memory access operations. Whilst the original implementation allows one to post-process data in-situ, we present in this paper extensions to the driver that provide the ability to couple parallel applications bidirectionally. We use this to perform computational steering of simulations. Commands and data are sent back to the simulation using either the driver layer itself (primarily commands) or the HDF5 layer via the DSM driver (datasets). The use of HDF5 datasets for transfer between codes makes true parallel coupling possible, even when the data models of the codes are not directly compatible. The steering interface presented here is shown implemented within ParaView, the parallel visualization application, but the API is generic and in fact any applications that make use of HDF5 may be connected using the driver